Image processing method and apparatus thereof

ABSTRACT

In case of an image region having a property of abrupt changes in luminance and tint, even when a region made up of pixels having luminance values and tint levels similar to those of one point designated by the user is extracted as a correction region with reference to the user designated point, it is difficult to extract a region to be corrected without omission. To solve this problem, a user instruction indicating a point inside or near an image region which is to undergo correction is input, and a plurality of origin pixels corresponding to start points of region expansion are set in a region which includes the input point and has a predetermined size. The region expansion is executed from each origin pixel, and a correction region is decided according to the result of the region expansion.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to image processing for detecting an imageregion that exhibits some poor color tone of an eye image.

2. Description of the Related Art

A method of correcting the poor color tone of an eye image due to aphotographic flash of a camera has been proposed. Note that the poorcolor tone of an eye image is generally well known as a red-eye effect.The red-eye effect is as follows. That is, upon photographing a personor an animal such as a dog, cat, or the like using a photographic flashunder an insufficient illumination environment, the flash light that hasentered the opening pupil part is reflected by the eyeground, and thecapillaries gleam red. Since a person who has a pale pigment color (or alight pigment) has a higher transmittance of the pupil, i.e., crystallens, the red-eye effect tends to occur more frequently with suchperson.

In recent years, digital cameras are increasingly downsized, and theoptical axis of a lens tends to be near the light source position of aphotographic flash. In general, as the light source position of thephotographic flash is closer to the optical axis of the lens, thered-eye effect occurs more readily, and a measure against thisphenomenon is a critical issue.

As one means for preventing the red-eye effect, a method of performingpre-light emission before photographing, and taking a picture after thepupil of the object is closed is known. However, this method consumes abattery more than normal photographing, and changes the expression ofthe object due to pre-light emission.

Many methods of compensating for a red-eye image by correcting andmodifying digital image data photographed by a digital camera using apersonal computer have been proposed.

The methods of correcting the red-eye effect on digital image data areroughly classified into manual correction, semiautomatic correction, andautomatic correction.

With the manual correction method, the user designates and corrects ared-eye region displayed on a display using a pointing device such as amouse, stylus, tablet, or the like, or a touch panel.

With the semiautomatic correction method, the user roughly designates aregion where a red-eye image exists, and correction is applied byspecifying a correction range of the red-eye image based on thatinformation. For example, the user designates a region that surroundsthe two eyes using a pointing device, or designates one point near theeye. Based on information of this designated region or designated point,the correction range is specified to apply correction.

With the automatic correction method, a correction region isautomatically detected from digital image data and correction processingis executed without any specific operation by the user.

Upon examining these correction methods, the automatic correction methodwhich does not require any troublesome operations for the user and canbe executed using even a device which does not comprise anyhigh-performance display with high visibility seems most useful.However, it is not guaranteed for the automatic correction method todetect a red-eye region from an image without fault, and a red regionother than a red-eye image may be erroneously detected and corrected.For this reason, image edit software or the like mounts a function ofcorrecting a red-eye image by combining automatic correction and manualor semi semiautomatic correction. The user applies automatic correctionto an image from which a red-eye image is to be corrected, and performsmanual correction if he or she is dissatisfied with that result. Inother words, the manual correction is an important function thatcompensates for the automatic correction precision.

The following techniques have been disclosed about the manual correctionand semiautomatic correction methods.

A technique disclosed by U.S. Pat. No. 5,130,789 makes the userdesignate a region including a red-eye image to be corrected, and alsodesignate one point in the red-eye image. The technique sets the colorof a pixel of this point as a target color, and generates a geometricshape to have the target color as the center. The technique then checksfor each pixel in the region designated by the user if the shapeincludes the color of an interest pixel, and determines pixels whosecolors are included in the shape as those which form the red-eye image(to be referred to as “red-eye pixels” hereinafter), thus correctingthese pixels.

Also, a technique disclosed by Japanese Patent Application Laid-Open No.6-350914 makes the user designate an outline of a correction region orone point near the correction region using a light pen or the like. Ifthe values of color components (e.g., RGB components) of an interestpixel included in the correction region fall within a predeterminedrange, the technique determines that the interest pixel is a candidatepixel which forms the red-eye image, and then checks if a set (region)of candidate pixels is a red-eye image. Then, the technique corrects theset (region) of candidate pixels determined as the red-eye image toblack.

When the user designates a region including an eye, a techniquedisclosed by Japanese Patent Application Laid-Open No. 10-75374 checksbased on the hue and saturation values of the designated region if ared-eye image exists in that region. If it is determined that a red-eyeimage in fact exists, the technique corrects that region. Furthermore,the technique detects catch light of an eye region, i.e., a light spotwhere flash light is reflected, and emphasizes the catch light if it isdetermined that the catch light is weak.

A technique disclosed by Japanese Patent Application Laid-Open No.2003-304555 makes the user designate an image region including an eye,and also at least one point in the red-eye image. The technique comparesthe red levels of neighboring pixels in the region designated by theuser, and extracts one or a plurality of red boundaries. The techniquechecks if the designated point falls within the red boundary. If thedesignated point does not fall within the red boundary, the techniquedoes not determine the interior of the boundary as a red-eye image. Thatis, this method allows manual correction with high certainty.

According to the research conducted by the present applicant, although ared-eye image is a local region, it has a broad luminance distributionfrom a low luminance level to a high luminance level, and its tint alsoranges from high-saturation red to low-saturation red. That is, thered-eye image has the property of abrupt changes in luminance and tintalthough it appears locally. The aforementioned technique determinesbased on the pixel of one point designated by the user a criterion todetermine whether or not the interest pixel is a red-eye pixel. In otherwords, the above technique extracts pixels having a luminance and tintsimilar to those of one point designated by the user as red-eye pixelsusing that point as a criterion. However, it is difficult to extract thepixels of the red-eye image having the above property using one point asa criterion.

The above technique uses the distance between the tint of an interestpixel and a target color. However, based on such simple criterion, it isdifficult to reliably extract the pixels of the red-eye image with theabove property.

Furthermore, the above technique uses hue and saturation to determinewhether or not the interest pixel is a red-eye pixel. However, thismethod suffers the following problems.

As is well known, when a pixel value is given by an RGB system, asaturation value S is given by:

S={max(R,G,B)−min(R, G, B)}/max(R,G,B  (1)

where max(R, G, B) is the maximum value of RGB components, and

min(R, G, B) is the minimum value of RGB components.

For example, as is apparent from the experimental results, the fleshcolor regions of Japanese people are normally distributed around 0 to30° in the hue domain (0 to 359°). Note that a hue angle of an HSIsystem near 0° is red, and it becomes more yellowish with increasing hueangle. The magnitude relationship of RGB values around 0 to 30°satisfies:

R>G>B  (2)

As described above, in case of a person who has a dark pigment color (ora rich pigment), a bright red-eye image hardly tends to form compared toa person who has a pale pigment color (or a light pigment).

In consideration of these facts, the red-eye pixel values and the pixelvalues of the flesh color region around the eyes of the Japanese can beestimated as:

Red-eye region: (R, G, B)=(109, 58, 65)

Flesh color region: (R, G, B)=(226, 183, 128))

In the above case, the saturation value of red-eye pixels is 40, andthat of pixels of the flesh color region is 43, i.e., these saturationvalues are roughly equal to each other. In other words, if pixels whichhave saturation values within a given range having the saturation valueof the target color as the center are determined as red-eye pixels, thepixels of the flesh color region are also more likely to be determinedas red-eye pixels.

SUMMARY OF THE INVENTION

The first aspect of the present invention discloses a method ofdetecting a correction region of an image, comprising the steps of:inputting a user instruction indicating a point inside or near a regionof an image which is to undergo correction; setting a plurality oforigin pixels corresponding to start points of region expansion in aregion which includes the input point and has a predetermined size;executing the region expansion from each of the origin pixels; anddeciding the correction region according to a result of the regionexpansion.

According to the present invention, an image region to be corrected canbe easily designated, and a correction region can be detected with highprecision. Also, a correction region of an image can be easily andaccurately detected.

Further features of the present invention will become apparent from thefollowing description of exemplary embodiments with reference to theattached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram showing the arrangement of a computer (imageprocessing apparatus) which executes image processing according to anembodiment of the present invention;

FIG. 2 is a functional block diagram showing an overview of correctionprocessing of a red-eye image according to the first embodiment;

FIG. 3 conceptually shows a red-eye image captured by an image sensingdevice such as a digital camera or the like;

FIG. 4 is a view showing a state in which the user designates one pointinside or near a region where he or she wants to correct a red-eyeimage;

FIG. 5 is a flowchart for explaining the processing of a correctionrange decision unit;

FIG. 6 shows an example of a region to be set in step S501;

FIG. 7 shows an example of an origin position to be decided in stepS503;

FIG. 8 shows an example in which top four points of color evaluationamounts E are set as origins;

FIG. 9 is a flowchart for explaining region expansion;

FIG. 10 shows an example of neighbor pixels;

FIGS. 11 and 12 are views for explaining processing for specifying acorrection region (red-eye region);

FIG. 13 is a view for explaining pixels which are registered in a list Lby region expansion;

FIG. 14 shows an example in which region expansion is made for aplurality of origins;

FIG. 15 is a flowchart showing correction processing for one of red-eyeregions decided as correction ranges;

FIG. 16 is a view for explaining processing for setting a correctionrange;

FIG. 17 is a view for explaining a method of setting correctionparameters;

FIG. 18 is a functional block diagram showing an overview of correctionprocessing of a red-eye image according to the second embodiment;

FIG. 19 is view showing a state in which the user designates a regionwhere he or she wants to correct a red-eye image by moving a mousepointer;

FIG. 20 is a flowchart showing the processing of a correction rangedecision unit;

FIG. 21 shows an example of a binary image obtained by binarization;

FIG. 22 shows an example of a user interface to be displayed on amonitor in the third embodiment;

FIG. 23 is a flowchart showing processing for determining whether theuser designates one point or a region;

FIG. 24 is a view for explaining a golden eye image;

FIG. 25 is a view for explaining correction of a golden eye region; and

FIG. 26 is a flowchart showing the processing of the fourth embodiment.

DESCRIPTION OF THE EMBODIMENTS

Image processing according to preferred embodiments of the presentinvention will be described in detail hereinafter with reference to theaccompanying drawings. Note that the image processing to be describedhereinafter is desirably incorporated in a printer driver whichgenerates image information to be output to a printer engine, and runsin a computer. Alternatively, the image processing to be describedhereinafter is desirably incorporated in a scanner driver which drivesan optical scanning device such as a scanner or the like, and runs in acomputer. Alternatively, the image processing to be describedhereinafter may be built in hardware such as a copying machine,facsimile, printer, scanner, digital camera, digital video camera, orthe like, or it may be supplied as software.

Embodiments for extracting a correction region will be describedhereinafter taking as an example a case wherein the poor color tone ofan image due to a red-eye effect is to be corrected. However, thepresent invention can extract a correction region other than a red-eyeregion by adjusting the calculation method of a color evaluation amount(to be described later) to an image region to be extracted.

First Embodiment [Arrangement of Apparatus]

FIG. 1 is a block diagram showing the arrangement of a computer (imageprocessing apparatus) which executes image processing according to anembodiment of the present invention.

A computer 100 comprises a CPU 101. The CPU 101 is connected to a ROM102 and RAM 103 via a system bus 109. Furthermore, the system bus 109connects a video card 104 that connects a monitor 113, a storage device105, an interface 108, and a network interface card (NIC) 107 to eachother.

The monitor 113 may comprise a touch panel. The storage device 105comprises a hard disk driver, memory card, or the like. A pointingdevice 106 includes a mouse, stylus, tablet, or the like. The interface108 is a serial bus interface such as USB (Universal Serial Bus),IEEE1394, or the like. To the interface 108, the pointing device 106, akeyboard 115, a printer 110, a scanner 111, a digital camera 112, andthe like can be connected. The NIC 107 establishes connection to anetwork 114.

The CPU 101 loads programs (including a program of image processing tobe described below) stored in the ROM 102 or storage device 105 onto theRAM 103 as a work memory, and executes the programs. By controlling theabove components via the system bus 109 in accordance with the programs,the CPU 101 implements the functions of the programs.

Note that FIG. 1 shows the general arrangement of hardware thatimplements image processing to be described in the first embodiment, andthe arrangement from which some components are omitted or to which otherdevices are added is included in the scope of the present invention.

[Overview of Processing]

FIG. 2 is a functional block diagram showing an overview of correctionprocessing of a red-eye image according to the first embodiment. The CPU101 executes this processing. Note that an input image is digital imagedata which is input from, e.g., the digital camera 112 or film scanner111 and is expressed by 8-bit RGB data, i.e., a total of 24 bits perpixel.

FIG. 3 conceptually shows a red-eye image captured by an image sensingdevice such as the digital camera 112 or the like, and illustrates apupil region 302 and iris region 301 of an eye, and a highlight region304 that forms due to photographic flash light used in photographing.Reference numeral 303 denotes a white portion. Normally, the pupilregion 302 becomes red due to a red-eye effect.

An input unit 202 shown in FIG. 2 receives image data from an inputterminal 201, and displays it on the monitor 113 so that the user canvisually confirm the input image. The user designates one point insideor near a region, where he or she wants to correct a red-eye image usingthe pointing device 106, with reference to the image displayed on themonitor 113. A correction position input unit 203 inputs this userinstruction.

FIG. 4 shows a state in which the user designates one point inside ornear a region where he or she wants to correct a red-eye image. The usermoves a mouse pointer 401 into the pupil region 302 of the red-eye imageand clicks a mouse button. The correction position input unit 203 inputsposition information on the image pointed by the mouse pointer at thetime of clicking.

A correction range decision unit 204 specifies the pupil region 302 ofthe red-eye image based on the position information input by thecorrection position input unit 203, and decides a correction range. Thecorrection range is decided by applying a region expansion method to theinput position information and grouping pixels having similar features,as will be described in detail later.

A correction unit 205 receives the correction range from the correctionrange decision unit 204, applies correction processing to thatcorrection range, and outputs the image data which has undergone thecorrection processing to an output terminal 206. The image data afterthe correction processing is displayed on the monitor 113 or is storedin the RAM 103 or storage device 105. Alternatively, the image data isprinted by the printer 110 connected to the interface 108 or istransmitted to another computer or server connected to the network 114(including an intranet and the Internet) via the NIC 107.

[Correction Range Decision Unit]

FIG. 5 is a flowchart for explaining the processing of the correctionrange decision unit 204.

The CPU 101 sets a region based on the position information input by thecorrection position input unit 203 (S501). FIG. 6 shows an example ofthe region to be set in step S501. The CPU 101 sets a rectangular region603 which has, as the center, a point 602 as the input positioninformation designated by the user, and a predetermined size. Therectangular region 603 can have an arbitrary size. For example, one sideof the rectangular region 603 is desirably set to be about severalpercentages (2 to 5%) of the short side of an image represented by theinput image data.

Next, the CPU 101 detects a pixel having a maximum color evaluationamount in the rectangular region 603 (S502). Note that the colorevaluation amount indicates the intensity of red calculated per pixel.The color evaluation amount can adopt various indices. For example, thesaturation of an HSI system, a Cr component after an RGB system isconverted into YCbCr as luminance and color difference components, andthe like may be used. Note that the first embodiment uses, as a specialevaluation amount, a red evaluation amount E given by:

E=(R−G)/R  (3)

Equation (3) means that the degree of redness of an interest pixel iscalculated from two, R and G components except for a B component inplace of the saturation level of a general HSI system. Advantagesobtained upon defining the red evaluation amount E by equation (3) inplace of saturation will be described below.

For example, in case of a person who has a dark pigment color (or a richpigment), since the transmittance of the crystal lens of the pupilregion 302 is low, a bright red-eye image hardly tends to form. Asdescribed above, experiments reveal that the estimated value of thepixel value of the red-eye region of the Japanese is (R, G, B)=(109, 58,65), and the hue values of the flesh color of many Japanese people aredistributed within the range from red (0°) to yellow (60°). In suchregion, R, G, and B components meet R>G>B, and the estimated value ofthe pixel value of the flesh color region around an eye is (R, G,B)=(226, 183, 128). That is, the pixels in both the red-eye region andthe flesh color region around the eye have small B components. In suchcase, the saturation value of the red-eye pixel is 40, and that of thepixel of the flesh color region around the eye is 43, i.e., thesesaturation values are roughly equal to each other. In other words, thesaturation value of the pixel of the red-eye region especially does notstand out with respect to that of the pixel of the flesh color regionaround the eye. Therefore, if the saturation value is used, it isdifficult to specify the correction range.

On the other hand, if the red evaluation amount E is defined by equation(3), i.e., as an evaluation amount which does not depend on the Bcomponent, the red evaluation amount E of the pixel of the red-eyeregion is 47, and that of the flesh color region around the eye is 19.That is, the red evaluation amount E of the pixel of the red-eye regionassumes a value twice or more that of the flesh color region around theeye.

For this reason, upon detecting a red-eye image of a person who has adark pigment color (or a rich pigment), since an evaluation amount isdefined by only R and G components without any B component, as given byequation (3), in place of the saturation, red-eye pixels can beextracted with high precision. In equation (3), the ratio of (R−G) to anR component defines the red evaluation amount E. However, the presentinvention is not limited to this. For example, (R−G) alone or R/G may beused as the red evaluation amount E.

Note that upon detecting a red-eye image of a person who has a palepigment color (or a light pigment), since an evaluation amount isdefined by only R and G components without any B component, as given byequation (3), in place of the saturation, red-eye pixels can beextracted with high precision, needless to say.

In FIG. 6, a pixel 601 has a maximum red evaluation amount E in therectangular region 603.

Next, the CPU 101 decides an origin position used in region expansion(S504) as the processing of the next stage (S503). The CPU 101 executesregion expansion to have this origin position as a start point. Forexample, it is an easiest way to set the user designated point 602 asthe origin position. Note that the first embodiment will explain amethod of region expansion from a plurality of origins.

FIG. 7 shows an example of origin positions to be decided in step S503.

The first embodiment sets a rectangle which has the user designatedpoint 602 and the pixel 601 with the maximum color evaluation amount Eas diagonal vertices, and decides the four vertices of the rectangle,i.e., the points 601 and 602 and points 702 and 703 as origins. That is,the region expansion (S504) executes expansion processing to have thefour points shown in FIG. 7 as start points.

Of course, the origins are not limited to the above four points. Of thefour points shown in FIG. 7, two points 702 and 703 may be used or threepoints excluding one arbitrary point may be used. Alternatively, the redevaluation amounts E of pixels in the rectangular region 603 arecalculated, and top several points of the color evaluation amounts E maybe used as origins. FIG. 8 shows an example in which top four points ofthe red evaluation amounts E (points 802 to 804 including the point 601)as origins. Furthermore, five points obtained by adding the userdesignated point 602 to the top four points may be used as origins.

Region Expansion (S504)

FIG. 9 is a flowchart for explaining region expansion.

The CPU 101 resets a parameter i to zero (S901), and registers thecoordinate position (pixel) of one origin at the i-th position (L[i]) ofa list L (S902). Note that the list L is an array which can store aplurality of coordinate positions.

The CPU 101 resets a parameter j to zero (S903), and calculatessimilarities of pixels (to be referred to as “neighbor pixels”hereinafter) which neighbor a pixel registered in L[j] (to be referredto as a “reference pixel” hereinafter) (S904). The CPU 101 checks if theneighbor pixels are similar to the reference pixel, and are included ina red-eye region (S905).

FIG. 10 shows an example of the neighbor pixels. Eight pixels whichneighbor a central reference pixel are set as neighbor pixels. Note thatnumerals given to boxes of the neighbor pixels are indices k of theneighbor pixels.

As the similarity, the CPU 101 calculates a difference R0 between a redevaluation amount Ej of the reference pixel and a red evaluation amountEk of the k-th (k=0 to 8) neighbor pixel:

R0=|Ek−Ej|  (4)

If the similarity R0 is smaller than a predetermined threshold T0, i.e.,if Ek and Ej are close to each other, the CPU 101 determines that thek-th neighbor pixel is similar to the reference pixel.

If R0<T0, similarity is high  (5)

Note that the similarity R0 may be defined using, e.g., a ratio betweenEk and Ej in place of their difference.

The technique described as the prior art calculates color evaluationamounts (e.g., hue values, saturation values, color differencecomponents, or RGB values themselves) between the neighbor pixels andreference pixel, determines their similarities based on the differences,and determines the neighbor pixel having a high similarity as a red-eyepixel. However, if such determination is made, the following problem isposed.

When the color evaluation amounts from pixel A to pixel C are calculatedand plotted for pixels which are located in a line segment-like region2501 shown in FIG. 11, a graph shown in FIG. 12 is obtained. That is,pixels from pixel A to pixel B are located within the pupil region of ared-eye image, and those from pixel B to pixel C are included in a fleshcolor region outside the red-eye image. According to the researchconducted by the present applicant, most of red-eye images have afeature that variations of color evaluation amounts in the red-eyeregion are larger than those of the flesh color region, as shown in FIG.12. Therefore, in order to extract red-eye pixels without omission, therange of color evaluation amounts to be detected as red-eye pixels,i.e., the range corresponding to the threshold T0, need be set as arange 1201 shown in FIG. 12. However, if such range is set, pixelsranging from pixel A to pixel C are extracted as red-eye pixels as wellas the flesh color region with less variations of color evaluationamounts, and the correction region includes the flesh color region. Inorder for the flesh color region to be excluded from the correctionregion, the range 1201 narrows down, and it becomes harder to extractred-eye pixels without omission.

In order to solve this problem, second determination is applied to theneighbor pixels which are determined to be similar to the referencepixel. The second determination will be described below incorrespondence with the example in FIG. 10. In the second determination,the CPU 101 calculates a difference Vj between the red evaluationamounts E of the reference pixel and the neighbor pixel (k=2)immediately above the reference pixel. Then, the CPU 101 calculates adifference Vk between the red evaluation amounts E of the neighbor pixelof k=2 and a surrounding pixel which further neighbors that neighborpixel. Then, the CPU 101 sets the difference between these differencesas a determination value R1 (see equations (6)). In this example, theCPU 101 calculates the determination value R1 between the neighbor pixelwhich is located immediately above the reference pixel and its upperneighboring surrounding pixel. Likewise, in case of a neighbor pixelwhich is located immediately below the reference pixel, the CPU 101calculates a determination value R1 between the neighbor pixel and itslower neighboring surrounding pixel. That is, the CPU 101 calculates adetermination value R1 from the red evaluation amounts E of three pixelswhich are contiguous in the upper direction, lower direction, leftdirection, right direction, upper left direction, upper right direction,lower right direction, or lower left direction to have the referencepixel as the center.

Vj=|Ek−Ej|

Vk=|En−Ek|

R1=|Vj−Vk|  (6)

where En is the red evaluation amount of a non-reference pixel whichneighbors the k-th neighbor pixel.

If Vj and Vk assume close values in step S905, in other words, if thesimilarity R1 of variation amounts is smaller than a predeterminedthreshold T1 (if inequality (7) holds), the CPU 101 determines that thesimilarity between the variation amounts of the interest pixel andreference pixel is high.

R1<T1  (7)

If such similarity determination of variation amounts is applied to thevariations shown in FIG. 12, since the variation characteristics of thered evaluation amount E of pixel B are apparently different from thoseof the red evaluation amounts E of surrounding pixels, the probabilityof erroneous extraction of pixels B to C can be reduced.

In this way, the second determination is further applied to the neighborpixels which are determined by the first determination to be similar tothe reference pixel. That is, for the neighbor pixels which have clearedthe first determination, whether or not the neighbor pixel of interestfalls within the red-eye region is determined based on the differencesVj of the red evaluation amounts from the reference pixel, and thedifferences Vk of the red evaluation amounts with non-reference pixelswhich neighbor in any of the left, right, upper, lower, and obliquedirections. In this way, the probability of extraction of pixels withinthe range from pixel B to pixel C as red-eye pixels due to thevariations of the red evaluation amounts E shown in FIG. 12 is reduced.That is, in addition to determination as to whether or not the redevaluation amount E is close to a reference value, whether or not aninterest pixel is included in the red-eye region is determined inconsideration of the degree of change in red evaluation amount E fromneighbor pixels, thus specifying the red-eye region with higherprecision.

In this way, the CPU 101 registers the neighbor pixels which havecleared the first and second determination processes in the list L(S906), and checks other neighbor pixels in the checking processing instep S907. Note that pixels of the neighbor pixels which have alreadybeen registered in the list L need not undergo the above determinationprocesses again.

In the above description, the determination value R1 is defined as thedifference between Vk and Vj. For example, the determination value R1may be defined using the ratio between Vk and Vj. Also, Vk is defined asthe difference between the red evaluation amounts E from thenon-reference pixel which neighbors the neighbor pixel in any of theright, left, upper, lower, and oblique directions. Alternatively, thedifference sum of the color evaluation amounts E of a plurality ofneighbor pixels without the reference pixel may be used.

Upon completion of determination of all the neighbor pixels of a givenreference pixel, the CPU 101 increments the parameter j (S908) andcompares the parameters i and j (S909). If i>j, the process returns tostep S904 to repeat determination for the next reference pixel L[j]. Ifi=j, since the determination for all the pixels registered in the list Lis complete, the region expansion processing ends.

FIG. 13 is a view for explaining pixels registered in the list L by theregion expansion. As a result of the region expansion with respect to anorigin pixel 1102, a pixel group 1101 is registered in the list L. Theregion expansion process allows to appropriately extract pixels in thepupil region 302 in most of red-eye images. However, in some red-eyeimages, only partial pixels of the pupil region 302 may be extracted, asshown in FIG. 13. In such case, if only the origin pixel 1102 and pixelgroup 1101 are used as the correction region, a desired correctionresult cannot be obtained.

In order to expand the pixel group 1101 to the entire pupil region 302,limitations using thresholds T0 and T1 may be relaxed. However, if thelimitations are relaxed, the region may be unwantedly expanded to theflesh color region. Hence, in place of relaxing the limitations, aplurality of origins of the region expansion are set, thus preciselyspecifying red-eye regions from the pupil region 302. Note that as thedecision method of a plurality of origins, the four vertices of arectangle having the pixel 601 with the maximum color evaluation amountE may be used as origins, as described above. Alternatively, the colorevaluation amounts E of pixels within the region set in step S501 may becalculated, and top several points of them may be used as origins.

FIG. 14 shows an example of region expansion for a plurality of origins.In FIG. 14, pixel groups detected based on region expansion from fourorigins exist. In step S506 shown in FIG. 5, the CPU 101 decides aregion 1202 which includes all pixels detected by the region expansionas the correction region. Note that FIG. 14 illustrates the region 1202as a rectangular region. Alternatively, a circular region, ellipticregion, or polygonal region may be used, or the user may select or inputthe shape of the region.

[Correction Unit]

The correction unit 205 corrects the correction range (red-eye region)decided by the correction range decision unit 204. FIG. 15 is aflowchart showing the correction processing one of red-eye regionsdecided as the correction ranges. The correction unit 205 receives thelist that describes one or more red-eye regions from the correctionrange decision unit 204, and applies correction processing to thered-eye regions one by one.

The CPU 101 decides a correction range for an interest red-eye region(S1301). FIG. 16 is a view for explaining the setting of the correctionrange. Referring to FIG. 16, a central rectangular region 1401 is thered-eye region described in the list. The CPU 101 sets, as a correctionregion, an elliptic region 1402 which has a major axis L_(H1) and minoraxis L_(H1) that pass through the center (to be referred to as “origin”hereinafter) of the red-eye region 1401.

L _(W1) =L _(W0) ×CPARAM _(—) AREARATIO

L _(H1) =L _(H0) ×CPARAM _(—) AREARATIO  (8)

where L_(W0) and L_(H0) are ½ the width and height of the red-eye region1401, and

CPARAM_AREARATIO is a parameter used to determine the correction range(for example, 1.2 to 1.5).

Next, the CPU 101 calculates a maximum luminance value Ymax and amaximum value Emax of the red evaluation amounts E in the ellipticregion 1402 as parameters required to correct the interior of theelliptic region 1402 (S1302).

The CPU 101 then checks if an interest pixel exists within the ellipticregion 1402 (S1303). Whether or not the interest pixel exists within theelliptic region 1402 is determined using the following inequality forcalculating an ellipse:

(x/L _(W1))²+(y/L _(H1))²≦1  (9)

If the interest pixel located at the coordinate position (x, y)satisfies inequality (9), the interest pixel exists within the ellipticregion 1402. The square roots in the left-hand side of inequality (9)correspond to the ratios of the distances from the origin to theinterest pixel to those from the origin to the circumference. If theinterest pixel does not exist within the elliptic region 1402, the CPU101 selects the next interest pixel.

If the interest pixel exists within the elliptic region 1402, the CPU101 converts the RGB values of the interest pixel to YCC values asluminance and color difference components (S1304). Various conversionmethods are available, and any of such methods may be used.

The CPU 101 calculates respective evaluation amounts for the interestpixel (S1305). The evaluation amounts are parameters required to decidea correction amount, and are the following three values:

Rr: a ratio r/r0 of a distance r from the origin to the interest pixelto a distance r0 from the origin to the circumference of the ellipse;

Er: a ratio Er/Emax of the red evaluation amount E of the interest pixelto Emax; and

Yr: a ratio Y/Ymax of a luminance value Y of the interest pixel to Ymax.

The CPU 101 calculates correction amounts Vy and Vc respectively for theluminance and color difference components of the interest pixel usingthe above parameters (S1306):

Vy={1−Rr ^(Ty1)}·{1−(1−Er)^(Ty2)}·{1−Yr ^(Ty3)}

Vc={1−Rr ^(Tc1)}·{1−(1−Er)^(Tc2)}  (10)

Both Vy and Vc assume values falling within the range from 0.0 to 1.0,and mean that the correction amounts are larger as they are closer to1.0. Vy is decided using all the three parameters, and becomes smalleras the position of the interest pixel is separated farther away from theorigin. When the red evaluation amount E of the interest pixel issmaller than Emax, Vy assumes a small value. When the luminance value Yof the interest pixel is close to Ymax, Vy assumes a small value.Setting a small correction amount of a high-luminance pixel provides aneffect of preserving a highlight part (catch light) in the eye. On theother hand, Vc is decided by excluding the evaluation amount associatedwith luminance from the equation of Vy.

In equations (10), Ty1, Ty2, Ty3, Tc1, and Tc2 are also parameters.Therefore, by setting these parameters, respective evaluation amounts(i.e., those in braces in equations (10)) can be applied as primary(solid line), secondary (broken line), and ternary (one dashed chainline) straight lines or curves, as shown in FIG. 17.

The CPU 101 calculates YCC values after correction using the correctionamounts Vy and Vc (S1307):

Y′=(1.0−Wy·Vy)·Y

C′=(1.0−Wc·Vc)·C  (11)

where Y and C are values before correction,

Y′ and C′ are values after correction, and

Wy and Wc are weights (0.0 to 1.0).

The weights Wy and Wc are adjusted when a correction strength is to bedesignated. For example, when three correction strength levels, i.e.,weak, medium, and strong, are to be set, results with differentcorrection strengths can be obtained by the same processing by settingboth Wy and Wc to 0.3, 0.7, 1.0, or the like.

After new values of the luminance and color difference components aredecided, the CPU 101 converts the YCC values into RGB values, andoverwrites the RGB values on a buffer memory for an input image as pixelvalues after correction, or stores them at predetermined addresses on abuffer memory that stores an output image (S1308).

Until it is determined in step S1309 that the last pixel correspondingto the interest red-eye region is reached, the CPU 101 shifts theinterest pixel in step S1303, and repeats the above processes (S1303 toS1308). If the last pixel corresponding to the interest red-eye regionhas been reached, the CPU 101 starts correction processing of the nextred-eye region, thus repeating the correction processing for all red-eyeregions recorded in the list.

Note that in the method described above, an image input to thecorrection unit 205 is made up of RGB components, and after the inputimage is converted into luminance and color difference components, whichare corrected, the corrected luminance and color difference componentsare converted again to RGB components. However, the present invention isnot limited to such specific method. For example, the RGB components maybe converted into lightness and saturation components, and after theselightness and saturation components are corrected using the same method,the corrected lightness and saturation components may be converted intoRGB components again, thus obtaining substantially the same result.

Furthermore, in the aforementioned example, the ratio Er of the redevaluation amount E of the interest pixel to the maximum amount Emax ofthe red evaluation amounts in the elliptic region 1402 is used as theparameter used to decide the correction amount. Alternatively, theparameter may be simply replaced by a saturation value. That is, thecorrection amount may be decided using the ratio of the saturation valueof the interest pixel to the maximum saturation value in the ellipticregion 1402.

As described above, since a plurality of origins of the region expansionfor extracting red-eye pixels are set, and the region expansion is donein consideration of variations of evaluation amounts with the neighborpixels, a red-eye region to be corrected can be decided with highprecision. Since the red evaluation amount E is calculated from two, Rand G components except for the B component in place of a saturationvalue of a general HSI system, a red-eye region can be decided with highprecision without being influenced by a pale pigment color (or a lightpigment) or dark pigment color (or a rich pigment).

Second Embodiment

Image processing according to the second embodiment of the presentinvention will be described below. Note that the same reference numeralsin the second embodiment denote the same parts as in the firstembodiment, and a detailed description thereof will be omitted.

The first embodiment has explained the method of specifying a correctionrange (red-eye region) by obtaining a plurality of origins based on theuser designated position, and performing region expansion from theseorigins. The second embodiment will explain a method of specifying acorrection range from a rectangular region designated by the user.

FIG. 18 is a functional block diagram showing an overview of processingfor correcting a red-eye image according to the second embodiment. TheCPU 101 executes this processing.

FIG. 19 shows a state in which the user designates a region 1702 wherehe or she wants to correct a red-eye image by moving a mouse pointer1703. The user designates the region 1702 by moving the mouse pointer1703 to one point (e.g., a point 1701) of the region, dragging the mousepointer 1703 while holding down a mouse button, and then releasing themouse button. Note that the region to be designated is not limited to arectangle, but a circle, an ellipse, or a polygon (defined when, forexample, the user traces the outer circumference of the region to becorrected) may be designated.

FIG. 20 is a flowchart for explaining the processing of the correctionrange decision unit 204.

The CPU 101 calculates an average amount Eave of the red evaluationamounts E of pixels in the rectangular region 1702 (S1801), andbinarizes the red evaluation amounts E of pixels in the rectangularregion 1702 using the average amount Eave as a threshold (S1802). Notethat the binarization threshold may use the average amount Eave intact,or may assume an amount obtained by adding or subtracting a certainmargin to or from the average amount Eave.

FIG. 21 shows an example of a binary image obtained by binarization.Upon binarizing the rectangular region 1702, pixels which have redevaluation amounts E larger than the threshold (indicated by □ in FIG.21; to be referred to as “candidate pixels” hereinafter) remain. The CPU101 stores this binary image in the RAM 103 or the like.

The CPU 101 then calculates the barycenter of the binary image (S1803),and calculates an average distance λ of the candidate pixels from thebarycenter (S1804). Then, the CPU 101 decides, as a correction range, acircular region 1902 shown in FIG. 21 which has the barycenter as thecenter and the average distance λ as a radius (S1805).

As shown in FIG. 21, in the binarization result using the average amountEave of the red evaluation amounts, candidate pixels are not alwaysconcentrated in the pupil region 302, and candidate pixels 1901 may alsoexist in the flesh color region. All candidate pixels in the pupilregion 302 are not always coupled, and the highlight region 304 in thepupil region 302 has a high possibility of being rejected from thecandidate pixels. Therefore, by calculating the aforementioned averagedistance λ of the candidate pixels from the barycenter, the pupil region302 can be specified with high precision.

As described above, when the user designates a region where he or shewants to correct a red-eye image, the correction range (red-eye region)can be decided with high precision.

Third Embodiment

Image processing according to the third embodiment of the presentinvention will be described below. Note that the same reference numeralsin the second embodiment denote the same parts as in the first andsecond embodiments, and a detailed description thereof will be omitted.

The first embodiment specifies a red-eye region based on the userdesignated point, and the second embodiment specifies it based on theuser designated region. However, these methods have both merits anddemerits. For example, designation of one point of the first embodimentrequires only a very simple operation since the user need only designateone point near a red-eye image. However, since a correction range isautomatically set, a correction range that the user did not actuallyintend to set may be set. The region designation of the secondembodiment can accurately designate a correction range that the userintended to set. However, the user must designate a region by mouseoperations and the like, and such operations are troublesome for userswho are unaccustomed to computers.

The third embodiment will explain a correction method that uses bothdesignation of one point and region designation.

FIG. 22 shows an example of a user interface 2201 displayed on themonitor 113 in the third embodiment. A window 2202 is used to designatea correction region by the user, and displays an image to be corrected.The user designates a desired correction region by operating a mousepointer 2203, and presses an “execution” button 2204 upon completion ofdesignation of the correction region, thus instructing to startexecution of correction processing. The user can select one pointdesignation or region designation upon designating the correctionregion.

FIG. 23 is a flowchart showing processing for determining if the userdesignates one point or a region. The CPU 101 executes this processing.

The CPU 101 checks if the mouse pointer 2203 is located on the window2202 and the user holds down the mouse button (S2101). If the mousepointer 2203 is located on the window 2202 and the user holds down themouse button, the CPU 101 stores coordinates A of the mouse pointer 2203at that time in the RAM 103 (S2102). The CPU 101 then checks if the userreleases the mouse button (S2103). If the user releases the mousebutton, the CPU 101 stores coordinates B of the mouse pointer 2203 atthat time in the RAM 103 (S2104). Note that when the mouse pointer 2203moves outside the window 2202 during the processes of steps S2101 toS2103, the CPU 101 clears the coordinates stored in the RAM 103, and theprocess returns to step S2101.

The CPU 101 calculates a distance λab between coordinates A and B storedin the RAM 103 (S2105) and compares the distance λab with apredetermined distance λth (S2106). If λab<λth, the CPU 101 determinesthat the user designates one point (S2107); if λab≧λth, the CPU 101determines that the user designates a region (S2108).

The reason why the CPU 101 compares the distance λab between A and Bwith the predetermined distance λth will be explained below. Positiondetection of an optical sensor adopted in recent mice has highsensitivity, and a position can be detected with very high precision. Bycontrast, such mouse may detect minute hand vibrations, and may move thepointer by a small distance. That is, even when the user designates onepoint, the pointer moves slightly between clicks of the mouse button,and the CPU 101 may erroneously determine region designation. Hence, thedistance λab and predetermined distance λth are compared and if themoving distance is small (less than λth), the CPU 101 determines onepoint designation.

As the predetermined distance λth, a value which has the number ofpixels as a unit may be set on a coordinate system having the upper leftcorner of the monitor as an origin. For example, on a monitor which hasa display resolution of 1600×1200 dots, λth=5 is set. In this case, ifcoordinates A are (100, 100), and coordinates B are (103, 103), sinceλab=√(3²+3²)=4.2, the CPU 101 determines one point designation.

However, if λth is fixed by the number of pixels, the operationalfeeling differs depending on the resolution of the monitor. For example,on a monitor of 3200×2400 dots, five pixels correspond to a smalldistance of about 0.1% in the horizontal direction. On the other hand,on a monitor of 600×480 dots, five pixels are about 0.8% in thehorizontal direction, and do not mean a small distance. In considerationof such display resolution differences, λth should be set as a ratio tothe display resolution. For example, if λth is set to be 0.5% of thenumber of dots in the horizontal direction, it corresponds to 16 pixelsfor 3200 dots, and three pixels for 600 dots. In this manner, when λthis set in association with the display resolution, the user canexperience an operational feeling which is not influenced by the monitorresolution.

Fourth Embodiment

Image processing according to the fourth embodiment of the presentinvention will be described below. Note that the same reference numeralsin the fourth embodiment denote the same parts as in the first to thirdembodiments, and a detailed description thereof will be omitted.

As a red-eye correction function, a technique that combines an automaticdetection and correction function and manual correction (assume that themanual correction includes semiautomatic correction in the followingdescription) has been proposed. Various red-eye automatic detectionmethods have been proposed. If any of these methods is adopted, it isdifficult to attain a 100% detection ratio, and a region which is notcorrected or correction other than a red-eye region occurs. Hence, themanual correction is used together to compensate for the automaticdetection and correction function, and a red-eye region which cannot beautomatically detected can be corrected.

In such combined use of the automatic detection and correction function,and manual correction, it is effective to change a correction method,correction strength, and the like in accordance with the automatic andmanual modes. For example, correction of a golden eye image is known.

A golden eye effect is a phenomenon that a pupil region gleams gold inplace of red. In most of golden eye images, catch light due tophotographic flash light, which exists in the red-eye effect, does notexist. Upon applying the correction for lowering the luminance and colordifference components described in the first embodiment to a golden eyeimage, the luminance level of the pupil region (a region 2402 shown inFIG. 24) tends to be originally high, and a sufficient correction resultcannot be obtained unless the luminance lowering amount is larger thanthat for a red-eye image. If the correction amount is increased, aregion outside the pupil region becomes extremely dark, and asatisfactory result cannot often be obtained. Since no catch lightexists, a very unnatural correction result is obtained.

Therefore, an effective correction method for a golden eye image is asfollows. As shown in FIG. 25, the interior of the detected pupil region2402 is filled with a predetermined pupil color (e.g., black), and awhite region 2404 as catch light is added to the center of the pupilregion 2402. In other words, effective correction of a golden eye imageis to discard original image data and to generate image data of thepupil region 2402.

However, if such correction is applied as automatic correction,detection errors stand out. As described above, using any of theautomatic detection, a desired region cannot be extracted 100%. Forexample, lips, a flaring candle, an ear lobe, or the like may beextracted as a golden eye region. It is readily understood that when theabove correction is applied to the erroneously detected region,corrected portions painted in, e.g., black stand out compared to thecorrection for lowering the luminance and color difference componentsdescribed in the first embodiment.

Hence, in the red-eye image correction function which has both theautomatic detection and correction function and the manual correctionfunction, extreme correction like that of a golden eye image is limitedto manual correction. With the manual correction, since the userdesignates a desired region (or position), a region other than an eye isnever corrected.

FIG. 26 is a flowchart showing the processing of the fourth embodiment.The CPU 101 executes this processing.

The CPU 101 checks if the current correction mode is an automatic ormanual mode (S2301). If the current correction mode is an automaticcorrection mode, the CPU 101 performs automatic detection of a red-eyeregion (S2302), and corrects a red-eye image using the method describedin the first embodiment for a correction region (S2303). As for thered-eye region automatic detection method, various methods such as thosedescribed in the paragraphs of the prior arts, and the like have beenproposed. Any of these methods may be used.

On the other hand, if it is determined that the current correction modeis a manual correction mode, the CPU 101 decides a correction regionbased on user's designation (S2304). The correction region can bedecided using the method described in the above embodiments. The CPU 101then calculates evaluation amounts used to determine if the correctionregion is a red-eye region or golden eye region (S2305). Since thegolden eye image is brighter than the red-eye image, and suffers lessluminance change in the pupil region as its property, the averageluminance value in the correction region and the standard deviation (orvariance value) of luminance values are used as the evaluation amounts.

The CPU 101 compares the calculated evaluation amounts withpredetermined thresholds to see whether or not the correction region isa golden eye region (S2306). If the average luminance value is largerthan a predetermined threshold and the standard deviation of luminancevalues is smaller than a predetermined threshold, the CPU 101 determinesthat the correction region is a golden eye region.

If it is determined that the correction region is a golden eye region,the CPU 101 performs the aforementioned correction of a golden eye image(S2307); otherwise, the CPU 101 corrects a red-eye image using themethod described in the first embodiment for the correction region(S2303).

Note that the evaluation amounts of a golden eye region are not limitedto the average luminance value and standard deviation of luminancevalues. For example, the average value and standard deviation of Rcomponent values may be used in place of the luminance values.

Modification of Embodiments

In each of the above embodiments, the red evaluation amount E which doesnot use any B of RGB component values is defined as a color evaluationamount for each pixel. However, the present invention is not limited tosuch specific value. For example, a red evaluation amount E may bedefined by:

E=(i·R+j·G+k·B)/R  (12)

where coefficients i, j, and k are weights which can assume negativevalues and the coefficient k may be set to be zero or a value smallerthan the coefficients i and j, thus obtaining the same effects.

Alternatively, after a pixel value is converted into another color spacesuch as Lab, YCbCr, or the like, a red evaluation amount E may bedefined regardless of a blue component or by setting a small weight ofthe blue component.

Other Embodiment

The present invention can be applied to a system constituted by aplurality of devices (e.g., host computer, interface, reader, printer)or to an apparatus comprising a single device (e.g., copying machine,facsimile machine).

Further, the object of the present invention can also be achieved byproviding a storage medium storing program codes for performing theaforesaid processes to a computer system or apparatus (e.g., a personalcomputer), reading the program codes, by a CPU or MPU of the computersystem or apparatus, from the storage medium, then executing theprogram.

In this case, the program codes read from the storage medium realize thefunctions according to the embodiments, and the storage medium storingthe program codes constitutes the invention.

Further, the storage medium, such as a flexible disk, a hard disk, anoptical disk, a magneto-optical disk, CD-ROM, CD-R, a magnetic tape, anon-volatile type memory card, and ROM can be used for providing theprogram codes.

Furthermore, besides aforesaid functions according to the aboveembodiments are realized by executing the program codes which are readby a computer, the present invention includes a case where an OS(operating system) or the like working on the computer performs a partor entire processes in accordance with designations of the program codesand realizes functions according to the above embodiments.

Furthermore, the present invention also includes a case where, after theprogram codes read from the storage medium are written in a functionexpansion card which is inserted into the computer or in a memoryprovided in a function expansion unit which is connected to thecomputer, CPU or the like contained in the function expansion card orunit performs a part or entire process in accordance with designationsof the program codes and realizes functions of the above embodiments.

In a case where the present invention is applied to the aforesaidstorage medium, the storage medium stores program codes corresponding tothe flowcharts described in the embodiments.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

This application claims the benefit of Japanese Patent Application No.2005-322646, filed Nov. 7, 2005, which is hereby incorporated byreference herein in its entirety.

1-15. (canceled)
 16. An image processing apparatus comprising: anobtaining section configured to obtain first coordinates on image datainstructed by pressing an operational button, and second coordinates onthe image data instructed by releasing the operational button; acalculating section configured to calculate a distance between the firstand second coordinates obtained by the obtaining section; a comparingsection configured to compare the calculated distance with apredetermined threshold value; and a determining section configured todetermine that region designation using the first and second coordinatesis performed in a case where a comparison result indicates that thedistance is equal to or greater than the predetermined threshold value,and to determine that one-point designation is performed in a case wherethe comparison result indicates that the distance is less than thepredetermined threshold value.
 17. The apparatus according to claim 16,wherein the predetermined threshold value is determined in accordancewith a display resolution of a device which displays the image data. 18.An image processing method comprising using a processor to perform stepsincluding: obtaining first coordinates on image data instructed bypressing an operational button, and second coordinates on the image datainstructed by releasing the operational button; calculating a distancebetween the first and second coordinates obtained in the obtaining step;comparing the calculated distance with a predetermined threshold value;determining that region designation using the first and secondcoordinates is performed in a case where a comparison result indicatesthat the distance is equal to or greater than the predeterminedthreshold value; and determining that one-point designation is performedin a case where the comparison result indicates that the distance isless than the predetermined threshold value.
 19. A non-transitorycomputer readable medium storing a computer-executable program forcausing a computer to perform an image processing method, the methodcomprising: obtaining first coordinates of image data instructed bypressing an operational button, and second coordinates of the image datainstructed by releasing the operational button; calculating a distancebetween the first and second coordinates obtained in the obtaining step;comparing the calculated distance with a predetermined threshold value;determining that region designation using the first and secondcoordinates is performed in a case where a comparison result indicatesthat the distance is equal to or greater than the predeterminedthreshold value; and determining that one-point designation is performedin a case where the comparison result indicates that the distance isless than the predetermined threshold value.